Digital cameras with triangulation autofocus systems and related methods

ABSTRACT

Digital cameras with triangulation auto focus systems include a spot beam emitter than can be used to project a spot onto a subject that is captured in an image and used to electronically automatically determine a subject to camera distance.

RELATED APPLICATION

This application is a continuation of U.S. patent application Ser. No.11/126,905, filed May 11, 2005 now U.S. Pat. No. 7,538,813, the contentsof which are hereby incorporated by reference as if recited in fullherein.

FIELD OF THE INVENTION

The present invention generally relates to digital camera devices, andmay be particularly suitable for mobile electronic devices that includea digital camera.

BACKGROUND OF THE INVENTION

Many digital still image and video cameras employ a passive type autofocus system that measures the contrast of image content. Generallydescribed, when comparing a sharp image with a blurred image, renderedon the same scene, the sharp image contains more information from highspatial frequencies. There are more transitions between dark and brightareas and the difference between the dark and bright areas is greater.Assessing the contrast of image content between images can give arelative measure of sharpness. A conventional contrast-based auto focussystem includes a camera lens, a focusing means for changing theposition of the lens, an imaging sensor and a digital signal processor(DSP). The lens is configured to move to different discrete focusingpositions, each corresponding to a particular subject-to-cameradistance. In operation, the system moves the lens to a number ofdifferent positions during the auto focusing process. An image iscaptured at each position and a relative sharpness value can bedetermined for each image. The system then moves the lens to theposition that produced the image having the greatest sharpness value.

A typical mobile phone digital camera uses the image contrast techniquefor auto-focusing. The mobile phone camera may cover a focusing distanceof from about 10 cm to infinity by using a resolution accomplished bybetween 10-20 focusing steps. Such a mobile phone camera may have a 2million pixel sensor and a fixed focal length lens with an aperturef1:2.8. This type of camera may produce about 15-30 images every secondin daylight or bright artificial light. It is believed that the autofocus actions for an image using these devices is relatively slow cantake about 1-2 seconds to perform.

Sub-sampling can be used to shorten this time. In the sub-samplingmethod, the lens does not move to every discrete location during thesearch process. The sharpness values can be interpolated to produce anestimation of sharpness for the positions between those actuallysampled.

Unfortunately, while sub-sampling may shorten the auto focus time, adegradation in precision may be introduced. In addition, the time ittakes to focus is still relatively long, typically about 1 second. Thismay make it difficult to achieve a good tracking auto focus for videorecording using the sub-sampling technique. For example, when the mainsubject is moving closer or farther away from the camera, the system canindicate that the sharpness has changed. However, it is difficult todetermine how much the subject has moved and/or in which direction. Somerelatively sophisticated and precise (relatively expensive) videosystems may address this problem by “hunting” the lens back and forth invery small steps.

There remains a need to provide alternative economic auto focus systemsfor digital cameras with still and/or video recording capability.

SUMMARY OF EMBODIMENTS OF THE INVENTION

Some embodiments of the present invention provide digital cameras withtriangulation auto focus systems. Certain embodiments of the presentinvention provide mobile communication devices incorporating digitalcameras that have triangulation auto focus systems.

Some further embodiments of the present invention are directed tomethods of auto-focusing a digital camera in still image and/or videorecording mode. The methods include: (a) projecting at least one lightspot toward a target subject; (b) capturing a first image of the targetsubject having the at least one light spot in response to the projectingstep; (c) programmatically determining a distance of the target subjectfrom the digital camera using the image with the light spot; and (d)automatically focusing a digital camera lens based on the determiningstep. In some embodiments, the methods can include: capturing a secondimage of a target subject devoid of an emitted light spot proximate intime to the capturing of the first image; electronically comparing aregion of interest in the first and second images to determinepositional data associated with the light spot in the first image; andelectronically calculating the distance of the target subject from thecamera based on the determined positional data. The first and secondimages can be captured using the same exposure setting and the samewhite balance setting.

The methods can be carried out so that the first and second images arecaptured within less than about 0.10 seconds of each other. In someembodiments, where the frame rate is between about 15-30 fps, theautomatically focusing step can be carried out in the range of about0.033-0.067 seconds. For increased frame rate speeds, the delay betweenthe captured images may be smaller than 0.033 seconds.

The methods can include auto focusing during video recording bymonitoring sequentially generated projected light spots in images; andelectronically removing the light spots from image data in the videorecording spot images.

In some embodiments the methods can include automatically stabilizingthe focus of a moving subject in a still image using the captured spotimage. The projecting step may include emitting light from a lightsource in a visible part of a light spectrum.

In some embodiments, the capturing step can include capturing an imagesignal using a digital camera sensor that is configured to integrate thesignal by detecting photons during image exposure time.

In some embodiments, the projecting at least one spot is carried out bysubstantially concurrently projecting a plurality of discrete spatiallyseparate spots. The methods can also include calculating a plurality ofsubject to spot distances, one distance for each of the plurality ofseparate spots. The automatically focusing step can use the spotcalculated as having a closest subject to camera distance to establish afocal distance for an image.

In some embodiments, the methods can include automaticallyelectronically registering the region of interest in the first andsecond images to thereby provide image stabilization.

Some further embodiments of the present invention are directed tomethods of providing motion image stabilization in images generated by amobile electronic communications device with a digital camera. Themethods include: (a) electronically capturing a reference image; (b)electronically capturing a target image with at least one spot imageregion; (c) automatically mathematically transforming at least the spotimage region of the target image by comparing at least a portion of thetarget image containing the spot image region to a corresponding regionin the reference image; and (d) generating a motion-stabilized imagebased on data from the mathematic transformation.

In some embodiments, the reference and target images can be capturedwithin less than about 0.10 seconds of each other (typically withinabout 0.033-0.067 seconds of each other). The methods may include:registering the target and reference images based on data from themathematic transformation; subtracting corresponding pixel values fromthe aligned target and reference images to provide a difference image;and determining the spot location based on the difference image. In someparticular embodiments, the subtracting corresponding pixel values stepcomprises separately accounting for different color pixel values in thecorresponding pixels.

Further embodiments of the present invention provide digital camerasconfigured with a triangulation auto focus system.

In some embodiments, the digital camera with the triangulation autofocus system forms part of a portable electronic communications device.

In some embodiments, the digital camera includes a spot beam emitter anda camera lens disposed proximate the spot beam emitter, the spot beamemitter being configured to project at least one spot onto a targetsubject that is able to be captured in a digital image of the targetsubject taken by the camera. The spot beam emitter may be configured toproject a light(s) that is in a visible part of the light spectrum.

The digital camera may include a camera image sensor in communicationwith the camera lens, the camera image sensor being configured to act asa receiver in the triangulation auto focus system.

In some embodiments, the digital camera can include a digital cameraimage sensor configured to communicate with the spot beam emitter and/oran automatic exposure control circuit in communication with the spotbeam emitter to control the duration and/or intensity of the projectedspot.

The digital camera may include a collimating lens in the lighttransmission path configured to collimate the light from the spot beamemitter

In some embodiments, the digital camera spot beam emitter is configuredto generate a plurality of spatially separate spots onto a subject thatis captured in the image.

In certain embodiments, the digital camera can include one or more ofthe following: a transparent diffraction grating, a cylindrical lens ora holographic diffraction grating, in communication with the spot beamemitter to substantially concurrently generate multiple spots in theimage.

The digital camera can include a spot beam emitter that is configured tooperate with a spot beam pulse having a spot duration of about 1 ms orless.

In some embodiments, the digital camera may further include a contrastauto focus system. The digital camera may include a light intensitycontrol circuit in communication with the spot beam emitter. The digitalcamera spot beam emitter may include a laser and the camera can includea pulse control circuit. The digital camera may include an automaticexposure control used to time a duration of the projected spot and tomonitor a brightness of a target image subject.

The digital camera may include a control circuit that is configured toselectively operate the camera in a still image camera mode and in avideo recording mode. The control circuit may be configured to controlemission of a spot beam pulse during the still image and video recordingmodes, and the control circuit may be configured to prevent emission ofthe spot beam pulse during acquisition of selected images.

The digital camera device may be configured to electronicallyautomatically compare a region of interest between electronically storedsequential first and second images to determine positional dataassociated with a light spot generated by the spot beam emitter, thenelectronically automatically calculate a distance of the target subjectfrom the camera based on the determined positional data and adjust theposition of the lens to the proper focal position based on thecalculated distance.

The digital camera device may be configured to automaticallyelectronically register and/or align a region of interest in the firstand second images to thereby provide motion image stabilization in astill image.

In particular embodiments, the digital camera is held in a portableelectronic communications device. The digital camera can include a spotbeam emitter configured to project a spot onto a target subject having atime duration and intensity sufficient to allow the spot to be capturedin a digital image of the target subject.

Other embodiments are directed toward computer program products foroperating a triangulation auto focus system in a digital camera. Thecomputer program product includes a computer readable storage mediumhaving computer readable program code embodied in the medium. Thecomputer-readable program code includes: (a) computer readable programcode that controllably operates a spot beam emitter to provide a spotthat is visible in a captured digital image; and (b) computer readableprogram code that calculates a subject to camera distance using adetermined spot distance of a spot from the spot beam emitter in thecaptured digital image.

The computer program product may also include computer readable programcode that automatically electronically registers and/or aligns a regionof interest between sequentially temporally proximate first and secondimages to thereby provide motion image stabilization in a still image.

Still other embodiments are directed to portable communications devicewith a digital camera comprising an auto focus system that is configuredto measure distance between a target subject and the camera using viewfinder image data and automatically move a camera lens to a focusposition based on the measured distance.

BRIEF DESCRIPTION OF THE DRAWINGS

Other features of the present invention will be more readily understoodfrom the following detailed description of specific embodiments thereofwhen read in conjunction with the accompanying drawings, in which:

FIG. 1 is a schematic illustration of a triangulation auto focus systemin accordance with various embodiments of the present invention;

FIG. 2A is a schematic illustration of a digital camera with an autofocus system with a spot beam emitter according to embodiments of thepresent invention;

FIG. 2B is a schematic illustration of a digital camera with an autofocus system having a collimating lens in communication with a spot beamemitter according to embodiments of the present invention;

FIG. 3A is a schematic illustration of a digital camera with an autofocus system using a spot beam emitter that projects a light spot onto atarget subject and a camera lens being configured to capture an image ofthe subject with the light spot according to embodiments of the presentinvention;

FIG. 3B is a schematic illustration of a digital camera with an autofocus system configured to substantially concurrently generate aplurality of projected light spots that can be captured in a digitalimage;

FIGS. 4A and 4B are front views of exemplary digital cameras accordingto embodiments of the present invention;

FIG. 5A is a block diagram of components of an auto focus system for adigital camera according to embodiments of the present invention;

FIG. 5B is a circuit diagram for controlling a pulse output of a laserspot beam emitter according to embodiments of the present invention;

FIG. 6 is a flow chart of operations that can be used for auto focusinga digital camera according to embodiments of the present invention;

FIG. 7 is a flow chart of operations that can be used to for autofocusing a digital camera according to embodiments of the presentinvention;

FIG. 8 is a flow chart of operations that can be used to carry out amethod of auto focusing a digital camera with motion stabilizationaccording to embodiments of the present invention;

FIG. 9 is a block diagram of a data processing system according toembodiments of the present invention;

FIG. 10A is a digital reference image of a target subject and FIG. 10Bis a spot image of the target subject, the images taken sequentially intime according to embodiments of the present invention.

FIG. 11A is a difference image of the reference and spot images shown inFIGS. 10A and 10B according to embodiments of the present invention;

FIG. 11B is a greatly enlarged image of the spot area in the differenceimage shown in FIG. 11A according to embodiments of the presentinvention;

FIG. 12A is a digital image of a target subject using a conventionalcontrast based AF system;

FIG. 12B is a digital image of the same target subject shown in FIG.12A, but taken with a multi-spot triangulating AF system according toembodiments of the present invention;

FIG. 13 is a QVGA-image with a plurality of spot positions which areenlarged for clarity in the extracted exploded portion shown to theright according to embodiments of the present invention; and

FIG. 14 is an example of source code that may be used toprogrammatically carry out image analysis calculations for evaluatingthe spot(s) position with image registration according to embodiments ofthe present invention.

DETAILED DESCRIPTION OF EMBODIMENTS OF THE INVENTION

The present invention now will be described more fully hereinafter withreference to the accompanying drawings, in which embodiments of theinvention are shown. However, this invention should not be construed aslimited to the embodiments set forth herein. Rather, these embodimentsare provided so that this disclosure will be thorough and complete, andwill fully convey the scope of the invention to those skilled in theart. Like numbers refer to like elements throughout. In the figures,certain layers, components or features may be exaggerated for clarity,and broken lines illustrate optional features or operations unlessspecified otherwise. In addition, the sequence of operations (or steps)is not limited to the order presented in the claims or figures unlessspecifically indicated otherwise. Where used, the terms “attached”,“connected”, “contacting”, “coupling” and the like, can mean eitherdirectly or indirectly, unless stated otherwise.

It will be understood that, although the terms first, second, etc. maybe used herein to describe various elements, these elements should notbe limited by these terms. These terms are only used to distinguish oneelement from another element. Thus, a first element discussed belowcould be termed a second element without departing from the scope of thepresent invention. In addition, as used herein, the singular forms “a”,“an” and “the” are intended to include the plural forms as well, unlessthe context clearly indicates otherwise.

It also will be understood that, as used herein, the term “comprising”or “comprises” is open-ended, and includes one or more stated elements,steps and/or functions without precluding one or more unstated elements,steps and/or functions. As used herein the term “and/or” includes anyand all combinations of one or more of the associated listed items. Itwill also be understood that when an element is referred to as being“connected” to another element, it can be directly connected to theother element or intervening elements may be present. In contrast, whenan element is referred to as being “directly connected” to anotherelement, there are no intervening elements present. It will also beunderstood that the sizes and relative orientations of the illustratedelements are not shown to scale, and in some instances they have beenexaggerated for purposes of explanation.

Unless otherwise defined, all terms (including technical and scientificterms) used herein have the same meaning as commonly understood by oneof ordinary skill in the art to which this invention belongs. It will befurther understood that terms, such as those defined in commonly useddictionaries, should be interpreted as having a meaning that isconsistent with their meaning in the context of this application and therelevant art and should not be interpreted in an idealized or overlyformal sense unless expressly so defined herein.

The term “electronic” means that the system, operation or device cancommunicate using any suitable electronic media and typically employsprogrammatically controlling the communication, interface protocol,timing and data exchange, and the like, between components of a digitalcamera with an auto focus system. The term “automatic” means thatsubstantially all or all of the operations so described can be carriedout without requiring active manual input of a human operator, andtypically means that the operation(s) can be programmaticallyelectronically directed and/or carried out.

The term mobile electronic (communication) devices includes portableradio communication equipment which can be describe a mobile radioterminal and includes all such equipment such as, for example, mobiletelephones, pages, communicators, electronic organizers, smartphones,hand held, laptop, or palm computers, personal digital assistants, andthe like.

The term “triangulation auto focus system” means an auto focus systemthat employs an active type of auto focusing that calculates a measureof distance between the camera and the subject rather than analyzingrelative sharpness by comparing multiple images. The triangulation autofocus system can project an emitted ray(s) of radiation from a lightsource integral to the camera toward the target subject. The projectedlight spot or spots can be detected in a digital image (typically inrelatively low resolution viewfinder image data) of the subject asreflected light. The term “spot” means a small, localized region zone ofan image that is due to the captured emitted ray(s) of radiation fromthe projected light spot. The spot can be any suitable shape, includinga strip or line, and can be sized to be detectable in the digital imageusing the resolution of the camera. In some embodiments, the spot can bebetween about 1-5 pixels, and in some particular embodiments, may be asub-pixel size. As will be discussed below, in some embodiments thedigital camera can be configured to generate multiple spots, one or moreof which can be used to determine the proper focal distance for thetarget subject.

Embodiments of the present invention can be used with any desiredresolution (higher resolution providing more detail). A typical standardsize/resolution for digital cameras, images (files), and displays is VGA(Video Graphics Array). VGA size is 640 pixels wide by 480 pixels tall(or vice-versa in portrait orientation). VGA has greater resolution thanCIF, QCIF, and QVGA, but smaller than SVGA, XGA, and megapixel. Inparticular embodiments, such as for compact mobile phones, the digitalcameras can be configured to provide QVGA (Quarter-VGA) having about 320pixels by 240 pixels which is larger (higher resolution) than QCIF, butsmaller than VGA.

As will be appreciated by one of skill in the art, embodiments of theinvention may be embodied as a method, system, data processing system,or computer program product. Accordingly, the present invention may takethe form of an entirely software embodiment or an embodiment combiningsoftware and hardware aspects, all generally referred to herein as a“circuit” or “module.” Furthermore, the present invention may take theform of a computer program product on a computer-usable storage mediumhaving computer-usable program code embodied in the medium. Any suitablecomputer readable medium may be utilized including hard disks, CD-ROMs,optical storage devices, a transmission media such as those supportingthe Internet or an intranet, or magnetic or other electronic storagedevices.

Computer program code for carrying out operations of the presentinvention may be written in an object oriented programming language suchas Java, Smalltalk or C++. However, the computer program code forcarrying out operations of the present invention may also be written inconventional procedural programming languages, such as the “C”programming language or in a visually oriented programming environment,such as VisualBasic.

The invention is described in part below with reference to flowchartillustrations and/or block diagrams of methods, systems, computerprogram products and data and/or system architecture structuresaccording to embodiments of the invention. It will be understood thateach block of the illustrations, and/or combinations of blocks, can beimplemented by computer program instructions. These computer programinstructions may be provided to a processor of a general-purposecomputer, special purpose computer, or other programmable dataprocessing apparatus to produce a machine, such that the instructions,which execute via the processor of the computer or other programmabledata processing apparatus, create means for implementing thefunctions/acts specified in the block or blocks.

These computer program instructions may also be stored in acomputer-readable memory or storage that can direct a computer or otherprogrammable data processing apparatus to function in a particularmanner, such that the instructions stored in the computer-readablememory or storage produce an article of manufacture includinginstruction means which implement the function/act specified in theblock or blocks.

The computer program instructions may also be loaded onto a computer orother programmable data processing apparatus to cause a series ofoperational steps to be performed on the computer or other programmableapparatus to produce a computer implemented process such that theinstructions which execute on the computer or other programmableapparatus provide steps for implementing the functions/acts specified inthe block or blocks.

Turning now to the figures, FIG. 1 illustrates a triangulation autofocus (AF) operational principle. The letters P₁, P₂ refer to subjectpositions. The AF system 10 includes a spot beam emitter 20 thatprojects the light beam 20 b toward the target subject (P₁ or P₂). Thespot beam emitter 20 can comprise an LED (light emitting diode), laseror other suitable light source. In some embodiments, the spot beamemitter 20 can operate in the visible portion of the light spectrum. Inother embodiments, the spot beam emitter 20 can be configured to emitlight that is in part of the light spectrum that is invisible to theeye. Combinations of these types of emitted light may also be used. Acamera sensor can then be used as the receiver 30 (as will be discussedfurther below).

The position of the spot image on the receiver surface 30 s shifts as afunction of subject distance as shown by the alternate position P₂ andassociated reflected beam 20 r′. The incoming receiver signal variesdepending on the position of the spot image. Spot position data can beextracted from the receive signal. Examples of some spot positiondetection methods that have been used in the past for other applicationsinclude solid state positioning sensing device (PSD), a slantedarrangement of dual photo diodes, a mechanically scanning photo diode, amechanically scanning LED, and a mechanically scanning obstructiondevice.

As shown in FIG. 1, the system 10 may include aspheric lenses, such as acollimating lens 40. The collimated light generates a projected light onthe subject. The system 10 can include a second lens 42 that projects animage of the subject on the surface of the receiver 30.

The spot signal should be isolated from signals originating from sourcesother than the spot beam emitter 20. For LED spot beam emitters 20 andcertain types of receivers 30, the reflected infrared radiation may beisolated by modulating the LED pulse with a certain frequency. Thesignal from the receiver 30 can then be filtered electronically tosuppress or remove unwanted or irrelevant signals.

As shown in FIGS. 2A and 2B, in some embodiments, a camera sensor 30 scan be used as the receiver 30. The system 10 can also include a cameralens 45 and a processor (such as a digital signal processor) with animage analysis module 50 and a lens positioner module 60 that define thesubject to camera distance and automatically electronically move thelens to the proper focal position. The lens 45 can be configured to moveto different discrete positions (such as for zone focusing) orcontinuous movement. As is known to those of skill in the art, insteadof capturing a photograph on film, as in earlier cameras, a digitalcamera, including those in a cameraphone, records the picture using asensor, a light-sensitive integrated circuit. The sensor is typicallyone of two main types, and these are known by the technology they use:either a CCD (charge-coupled device), which uses an array of capacitors,or a CMOS (complementary metal-oxide-semiconductor) device, which usestransistors. Whatever the type of sensor fitted, the principle ofoperation is generally the same: the light that passes through the lensfalls onto the sensor, which contains many individual light-sensitivecells, and each cell reacts to a particular point of light coming fromthe scene that is to be photographed. The effect of the incoming lightis to cause tiny changes in the individual cells of the sensor, creatinga pattern that mirrors the image, and these changes are then convertedinto electrical signals. The electrical output from the sensor is nextput through several stages of processing, so that the image of thephotographed scene can be reproduced on the viewfinder, or it may bestored in the phone's memory for later use.

The sensor 30 s may note be sufficiently sensitive to infraredradiation, so the spot beam emitter 20 can be configured to generatelight that is in a visible part of the light spectrum. The spot imagedata captured by the camera sensor 30 s can be processed to identify thelocation of the spot in the image.

In some embodiments, two sequential images can be compared to determinethe spot positional data. One of the images can be the spot image andthe other can be a “normal” image with the beam spot emitter beinginactive, with the normal image being devoid of a spot in the image. Thetwo images can be taken in any desired order (spot image first orsecond) and are generally taken relatively quickly within less thanabout 0.10 seconds of each other. A typical digital camera can operateat about 15-30 frames/second and the two images can be captured withinabout 0.033-0.067 of each other. To facilitate comparison of the images,the exposure and white balance may be locked at the same level for thetwo images.

Only a relatively small portion of the images, the portions in theregion of interest where the spot can be present need to be compared(saving on image processing time). The autofocusing can be carried outrelatively quickly in less than about 0.5 seconds, typically in lessthan about 0.25 seconds, even in total darkness.

FIG. 3A illustrates that that the lens axis L_(A) and the emitter axisE_(A) can be substantially parallel. In some embodiments the distancebetween the two “D” axes can be less than about 3 cm, typically lessthan about 1 cm. The spot beam emitter 20 can be located proximate thelens, and is typically located above or below the lens 45 as shown inFIGS. 4A and 4B (or both above and below where multiple spot beamemitters are used—not shown).

Typical mobile phone cameras have focal lengths of about 5 mm and anaperture of f1:2.8, distance information may only be desired where thesubject is located at about 2.7 meters or closer to the camera 10. Thatis, there may not be a detectable spot signal when the distance tosubject is long. The lens can be focused on the hyper focal distance inthese situations. Alternatively, the camera 10 may include both acontrast AF system and the triangulation system and may (automatically)employ the contrast AF system above defined distances. Because contrastAF systems do not require additional hardware, the cost of providingboth types of systems is relatively low.

FIGS. 4A and 4B illustrate that the camera 10 can include a view finder47 and may optionally also include a flash (such as a xenon flash)and/or a light sensor 49 and accompanying flash control circuit forfacilitating exposure (brightness) control. The camera 10 can have anautomatic exposure control that uses the light sensor 49 to generateinformation on the subject's brightness. The information can also beused to time the duration and/or intensity of the spot beam. In otherembodiments, a predetermined intensity can be set for operative use.Alternatively, the camera 10 can include a user adjustable intensitysetting. It is also noted that it may be possible to omit the lightsensor 49 because the distance information from the AF system providedby the instant invention can be used as a basis for exposure controlduring flash lighting images. FIG. 4B also illustrates that the camera10 can be included in a portable communications device 10 c.

FIG. 5A illustrates components of a digital camera 10. As shown, thespot beam emitter 20 can optionally be in communication with a lightintensity control circuit 23 to control and/or adjust the intensity ofthe spot beam. In any event, the camera 10 can employ a circuit 23 tocontrol the intensity and or time duration of the beam projected by thebeam spot emitter 20. For example, where the camera deploys a flash, thespot beam emitter 20 may employ a more intense spot beam 20 r. When thecamera circuit 23 detects a non-flash (non full sun) picture, the spotbeam emitter 20 may generate a less intense beam.

FIG. 5A also illustrates that the camera lens 45 is in communicationwith an electronic storage media for capturing the digital spot image 32and a subject focus distance calculation module using the spot positionin the spot image 51′. The distance calculation module 51′ can beconfigured as part of the image processing module 51 shown in FIGS. 2Aand 2B. The camera 10 also includes a lens position focus module 70 witha means for automatically moving the lens 45 to the proper focusposition as is well known to those of skill in the art. The camera 10can also include an amplification system for amplifying the receivedsignal (not shown).

Turning back to FIG. 3A, as discussed above, the spot beam emitter 20can comprise an LED typically with a collimating lens 40 (FIG. 1) orlaser. The camera 10 can include an electronic circuit for driving theLED (not shown). A laser (such as a solid state laser) may be used forfocusing at longer distances because it can operate at a higher lightpower and project a well-defined spot in the image, even at longerdistances.

In some embodiments, as illustrated in FIG. 3B, the system 10 can beconfigured to generate a plurality of discrete spots using a pluralityof projected beams, 20 b ₁, 20 b ₂. The plurality of spots can provide abroader AF area. The multiple spots can be generated in several manners.For example, discrete spot beam emitters as noted above. Alternatively,a single beam spot emitter, such as a laser 20 a, can be incommunication with an optic system 21 that splits the beam into discreterays. The optic system 21 can include a transparent diffraction gratingthat can generate spot beams that diverge and generate spot images witha constant separation in the receiver's image independent of subjectdistance. Another example of an optic system 21 is a cylindrical lensand/or holographic diffraction grating that that can produce ahorizontal line image. The multiple spots can be generally aligned inthe spot image (in a row and/or column), typically relatively centeredabout the center of the image.

For certain spot beam emitters 20 employing lasers, it may be desirableto keep laser power low, typically in the range of about laser pointersand to make the duration of the laser pulse short, such as less thanabout 2 milliseconds, typically less than about 1 millisecond, tocontrol the exposure to a subject's eye(s). The laser pulse can be timedand configured to be sufficiently short and with low energy so as toreduce the likelihood that the camera 10 may need laser notice orwarning labeling.

FIG. 5B illustrates an exemplary pulse control circuit 80 that can beconfigured to control laser pulse duration. In operation, current flowthrough the resistor 81 is not sufficient to power a laser spot beamemitter (20, FIGS. 3A, 3B). The capacitor 82 is charged through theresistor 81. When the capacitor 82 is charged, it can deliver sufficientenergy for one short laser pulse. The next laser pulse cannot be emitteduntil the capacitor is sufficiently charged again, which will take acertain time depending on the time constant τ (which is dependent on thevalue of R and C) and the threshold voltage of the laser.

FIG. 6 illustrates exemplary operations that can be used to carry outauto focusing digital cameras according to embodiments of the presentinvention. At least one light spot is projected toward a target subject(block 100). A first image of the target subject having the at least onelight spot is electronically captured (block 105). A distance of thetarget subject to the digital camera is programmatically determinedusing the image with the at least one light spot (block 110). Thedigital camera lens is automatically focused based on the determineddistance (block 115).

In some embodiments, the image of a moving subject may optionally bestabilized using the captured spot image (block 116). The stabilizationprocess can correct for any misalignments caused by hand shake of theuser or movements in the scene. Similarly, auto focusing during videorecording can optionally comprise monitoring sequentially generated spotimages and electronically removing light spots (such as by replacing oradjusting pixel values) from image data in the video recording spotimages (block 117).

In some embodiments, the AF system can find the position of the spots invideo recording images, because a spot can be visible every time an AFmeasurement is performed. Once the spots are recognized in the spotimage(s) they can be removed by patching image data from thecorresponding reference image(s). As such, the lens does not need to“hunt” for the best focus in video mode and tracking AF. The distancecan be checked at regular intervals and a correction of focus can bemade only when the system determines that new data is reliable and anadjustment is appropriate. Thus, the system can perform predictable AFwhen shooting still images or video sequences of moving objects. Themotion of the moving object can be predicted based n a series ofdistance measurements.

FIG. 7 illustrates operations that may also be used to implement aspectsof the invention. As shown, a second image of the target subject, thisone without the emitted light spot, can be captured proximate in time tothe capturing of the first image (block 125). The second image can becaptured before or after the first spot image (block 126). The twoimages can be electronically compared to determine positional dataassociated with the light spot (block 130). The distance of the targetsubject from the camera can be calculated based on the determinedpositional data (block 135) as the position of the spot correlates todistance.

The comparison may optionally include automatically electronicallyregistering (aligning) a region of interest in the first and secondimages to provide motion image stabilization, particularly in stillimages (block 131). The region of interest is less than the entire imageand is typically a very small portion of the image that is sized tosurround the spot zone(s).

FIG. 8 illustrates operations that can be used to electronicallyautomatically provide a motion-stabilized still image. The operationsmay be used separately or in combination with other methods of theinvention (as a motion-stabilization feature which does not require theuse of triangulation AF systems). A reference image and a target imagewith at least one spot region are electronically captured (block 140).The spot image region in the target image is mathematically transformedby comparing a corresponding portion of the reference image (block 145).A motion-stabilized still image is generated based on the transformation(block 150).

In some embodiments, the target and reference images are registeredbased on the mathematic transformation (block 146). Values ofcorresponding pixels can be subtracted from the registered (aligned)target and reference images to provide a difference image (block 147).The spot location can be determined based on the difference image (block148).

The transformation can shift the image horizontally and vertically aswell as rotated while comparing the reference image using a correlationratio algorithm. The correlation ratios for different transformationsmay be repeatedly calculated until the “optimal” or a sufficientlyacceptable transformation is determined. Because the transformation isperformed on a subset of the image, the difference between the images isexpected to be relatively small.

Thus, in certain embodiments, when the two images are registered, theircorresponding pixel values can be subtracted. The result is a differenceimage with the spot(s) against a dark background. Of course, the othercontrasts can be used. The difference image can be generated usingEquation (1).Dn=Xn−Yn   Equation 1where “Dn” represents a single pixel value of the difference imagebetween the corresponding pixel values of the spot image frame “Xn” andthe reference image frame “Yn.” Pixel values for different colors may betreated separately. For example, pixel values for red, green and bluecolors can be treated separately. The calculation can be repeated forall pixels in the region of interest. Finding the spot position in thedifference image may be more precise if the spot color is taken intoaccount. A second difference image (D2) can be calculated using Equation(2), which assumes that the spot beam emitter is a red emitting lightsource (e.g., LED). The difference image D2 is a monochrome image.Alternative equations can be used to emphasize the color characteristicof the spot(s).D2n=Xn[red]−Yn[red]  Equation 2where the pixel of D2 having the highest value corresponds to the spotposition in the image.

The distance between the camera and the subject can be calculated.Equation 3 is an example of an equation that may be used to calculatethe distance. The equation assumes that the axis of the camera lens isparallel to that of the emitter ray(s) or beam(s).u=fs/np   Equation 3where, the displacement (n) refers to the difference in position betweenthe spot image and the center of the image. The calculated subject tocamera distance (u) depends also on the focal length of the camera lens(f), the distance between the camera lens and the emitter axis (s), thedisplacement of the spot measured in number of pixels (n), and pixelpitch (p).

It is believed that accuracy may be further improved by applying apolynomial function to the pixel values representing the spot image inthe direction of interest, which may allow sub-pixel accuracy.

It will be appreciated that a spot signal may not be detectable when thedistance to the subject is very long. The lens should be focused on thehyper focal distance in such a situation. This should be satisfactorybecause no signal response can be assumed to correlate to a very longdistance. Alternatively, the contrast AF system may be employed in sucha situation.

In a typical mobile phone camera having a focal length of 5 mm and anaperture of f1:2.8, distance information may only be appropriate incases where the subject is located closer than about 3 meters (such asabout 2.7 meters or closer). The calculated subject distance correspondsto a certain focusing position of the camera lens. The lens is moved tothe correct position using the distance information. The triangulationmethod of determining distances is suitable for the auto-focusing of thedigital camera lens.

FIG. 9 illustrates an exemplary data processing system that may beincluded in devices operating in accordance with some embodiments of thepresent invention. As illustrated in FIG. 9, a data processing system,which can be used to carry out or direct operations includes a processor200, a memory 236 and input/output circuits 246. The data processingsystem may be incorporated in the digital camera and/or portablecommunication device, or the like. The processor 200 communicates withthe memory 236 via an address/data bus 248 and communicates with theinput/output circuits 246 via an address/data bus 249. The input/outputcircuits 246 can be used to transfer information between the memory(memory and/or storage media) 236 and another component. Thesecomponents may be conventional components such as those used in manyconventional data processing systems, and/or image processingcomponents, lens positioner, and the like, which may be configured tooperate as described herein.

In particular, the processor 200 can be commercially available or custommicroprocessor, microcontroller, digital signal processor or the like.The memory 236 may include any memory devices and/or storage mediacontaining the software and data used to implement the functionalitycircuits or modules used in accordance with embodiments of the presentinvention, The memory 236 can include, but is not limited to, thefollowing types of devices: cache, ROM, PROM, EPROM, EEPROM, flashmemory, SRAM, DRAM and magnetic disk. In some embodiments of the presentinvention, the memory 236 may be a content addressable memory (CAM).

As further illustrated in FIG. 9, the memory (and/or storage media) 236may include several categories of software and data used in the dataprocessing system: an operating system 252; application programs 254;input/output device drivers 258; and data 256. As will be appreciated bythose of skill in the art, the operating system 252 may be any operatingsystem suitable for use with a data processing system, such as IBM®,OS/2®, AIX® or zOS® operating systems or Microsoft® Windows®95,Windows98, Windows2000 or WindowsXP operating systems Unix or Linux™.IBM, OS/2, AIX and zOS are trademarks of International Business MachinesCorporation in the United States, other countries, or both while Linuxis a trademark of Linus Torvalds in the United States, other countries,or both. Microsoft and Windows are trademarks of Microsoft Corporationin the United States, other countries, or both. The input/output devicedrivers 258 typically include software routines accessed through theoperating system 252 by the application program 254 to communicate withdevices such as the input/output circuits 246 and certain memory 236components. The application programs 254 are illustrative of theprograms that implement the various features of the circuits and modulesaccording to some embodiments of the present invention. Finally, thedata 256 represents the static and dynamic data used by the applicationprograms 254 the operating system 252 the input/output device drivers258 and other software programs that may reside in the memory 236.

The data processing system 116 may include several modules, including aTriangulation AF module 220 (using a measure of distance to focus thelens), a spot position determination module 224, an image capture andregistration module 225, and the like. The modules may be configured asa single module or additional modules otherwise configured to implementthe operations described herein.

While the present invention is illustrated with reference to theapplication programs 220, 224, 225 in FIG. 9, as will be appreciated bythose of skill in the art, other configurations fall within the scope ofthe present invention. For example, rather than being an applicationprogram 254 these circuits and modules may also be incorporated into theoperating system 252 or other such logical division of the dataprocessing system. Furthermore, while the application programs 220, 224,234 in FIG. 9 are illustrated in a single data processing system, aswill be appreciated by those of skill in the art, such functionality maybe distributed across one or more data processing systems. Thus, thepresent invention should not be construed as limited to theconfigurations illustrated in FIG. 9, but may be provided by otherarrangements and/or divisions of functions between data processingsystems. For example, although FIG. 9 is illustrated as having variouscircuits and modules, one or more of these circuits or modules may becombined, or separated further, without departing from the scope of thepresent invention.

FIG. 10A is a digital reference image 300 of a target subject. FIG. 10Bis a digital spot image 310 of the same target subject. Both images havethe same size of 240 by 320 pixels (QVGA). In FIG. 10B, a small spot 311generated by the spot beam emitter is visible at a generally mediallocation in the spot image 310.

FIG. 11A is a difference image 320 of the spot image 310 and thereference image 300 shown in FIGS. 10B and 10A, respectively. FIG. 11Bis a difference image with the spot area 311 magnified. The differenceimage 320 was processed using the images and a computer program usingEquation (1). The triangulation AF algorithm works sufficiently well inQVGA resolution and can be used on viewfinder image data and does notrequire the camera to operate in high resolution mode.

FIG. 12A is a digital image 400 c taken using a conventional contrastbased AF system that sometimes has difficulty focusing on the correctpart of the image. In this example, the background contributes with moreimage detail than the bird in the foreground and the system can befooled to focus on the background. Thus, focusing problems can arisewhen wide focusing is applied to contrast AF systems. As shown, forexample, if the focusing area is covering a subject with sparse detailsin the foreground and the background contains a lot of details (such asa trees, wooded areas or a structure like a wall of a building), thecontrast system may decide to focus on the background rather than theforeground.

FIG. 12B illustrates a digital spot image 400 s taken using thetriangulation AF system. The use of a multi-spot pattern can allow forrelatively wide focusing using the triangular method and a plurality ofspatially discrete spots 411 As shown, the spots (shown as three, buttwo, or four or more may be used) are horizontally spaced about thecenter of the image. Vertical or angular spot patterns may also be used.Thus, the image 400 s can be segmented into multiple regions ofinterest, one for each spot in order to consider the multiple spots 411.Each region can be separately interrogated using the algorithms andequations described herein. A spot position for each spot 411 in eachregion of interest can be calculated. The spot position corresponding tothe lowest distance value can be used for focusing as the assumption canbe made that the part of the subject positioned closest to the camera isthe part of the scene where the user most likely wants to focus.

The triangulation AF system can distinguish between the subject locatedat different distances. In this example, only the central and rightspots will result in a detectable reflection, as the left spot isassociated with a further spot having an increased distance value.

Zone focus scale and depth of field can be carried out usingtriangulation data. The method of controlling the focusing action of thelens in discrete steps as opposed to continuous movement is usuallycalled “zone focusing”. The zone focusing is based on the depth of fieldfor the lens in use. A zone focusing scale can be calculated from aseries of fractions of the hyper focal distance. The hyper focaldistance is calculated using Equation 4.h=f ² /NC   Equation 4where “h” is the hyper focal distance, f is the focal length of thelens, N is the lens aperture and C is the circle of confusion (maximumallowed blur circle).

FIG. 13 is an illustration of spot images in a QVGA image with anenlarged spot portion exploded to the right of the image. In thisexample, the location of the spot beam emitter is assumed to be locatedunder the camera lens. Table 1 illustrates an example of a zone focusingscale and calculation of the pixel steps of spot position in thecaptured image shown in FIG. 13 are shown in Table 1. The distancesFOCUS, CLOSE and FAR are given in meters. In this example, the spots arelocated at the same horizontal pixel (5), and vary in vertical distancewith respect to center in steps of 3 pixels, from 1 to 43 pixels.

TABLE 1 Zone Focusing Table Using Spot Position Hyperfocal distanceFOCUS CLOSE FAR PIXELS STEP h 2.98 1.49 INF 1.5 3.0 h/3 0.99 0.74 1.494.5 3.0 h/5 0.60 0.50 0.74 7.5 3.0 h/7 0.43 0.37 0.50 10.5 3.0 h/9 0.330.30 0.37 13.5 3.0 h/11 0.27 0.25 0.30 16.5 3.0 h/13 0.23 0.21 0.25 19.53.0 h/15 0.198 0.186 0.213 22.5 3.0 h/17 0.175 0.165 0.186 25.5 3.0 h/190.157 0.149 0.165 28.5 3.0 h/21 0.142 0.135 0.149 31.5 3.0 h/23 0.1290.124 0.135 34.5 3.0 h/25 0.119 0.114 0.124 37.5 3.0 h/27 0.110 0.1060.114 40.5 3.0 h/29 0.103 0.099 0.106 43.5

Table 1 illustrates the relationship between the mathematics of focusingand triangulation. A zone focusing scale is constructed from the hyperfocal distance. A lens that is focused at the hyper focal distance willhave a depth of field ranging from h/2 to infinity. The following stepsof the scale are constructed by dividing “h” with odd integers. Any stepin the scale will have a depth of field reaching between the next andprevious steps. The column “PIXELS” shows how many pixels from thecenter of the image the spot will be visible at the given distance. Thecolumn “STEP” shows how many pixels there are between the next andprevious steps. The example is based on a digital camera having lenswith the focal length of 5 mm and speed 1:2,8. The camera sensor is a 2million pixel-array with a pixel pitch of 3 μm. The spot is detected inthe viewfinder image having the resolution of 240×320 (QVGA). Thedistance between the camera lens axis and the spot beam axis is 13, 67mm in this example. The steps between the spot positions measured inpixels are equidistant making the construction and mathematicalimplementation relatively straightforward. The step resolution of 3pixels provides sufficient resolution for this example.

FIG. 14 is an exemplary source code listing that can be used to carryout operations according to some embodiments of the present invention.This example of code does not include image registration. It is notedagain that only the region/regions of interest aresearched/interrogated. The code is configured to subtract the referenceimage from the spot image to determine the position of the spot.

In summary, some embodiments of the present invention can provide one ormore of increased auto focusing speed, image stabilization, can operatein total darkness, can make continuous tracking AF possible, can improvefocusing precision, can reduce cost over conventional triangulationsystems, can be implemented in relatively small size.

The present invention may also be embodied in other types of mobileelectronic apparatus that include a camera without departing from thescope of the present invention. For example, the mobile electronicapparatus may include a RF radio and a camera, and/or it may include amobile communication terminal such as a satellite or cellularradiotelephone and a camera device. The camera phone may be configuredto transmit pictures over a wireless interface, such as a cellular RFinterface, and may display the pictures on a display. The camera phonemay also be configured to generate an audio signal from communicationsignals that are received over a wireless interface. The transmission ofdigital pictures and generation of audio signals from receivedcommunication signal may be carried out by the camera phone in aconventional manner.

Many variations and modifications can be made to the preferredembodiments without substantially departing from the principles of thepresent invention. Accordingly, all such variations and/or modificationsare intended to be included within the scope of this invention asdefined in the claims. In the claims, means-plus-function clauses, whereused, are intended to cover the structures described herein asperforming the recited function and not only structural equivalents butalso equivalent structures. Therefore, it is to be understood that theforegoing is illustrative of the present invention and is not to beconstrued as limited to the specific embodiments disclosed, and thatmodifications to the disclosed embodiments, as well as otherembodiments, are intended to be included within the scope of theappended claims. The invention is defined by the following claims, withequivalents of the claims to be included therein.

1. A method of auto-focusing a digital camera in still image and/orvideo recording mode, comprising: projecting a plurality of discretelight spots toward a target subject using the digital camera; capturinga first viewfinder image of the target subject with the projected lightspots in response to the projecting step; programmatically calculating atriangulation measure of distance for each of a plurality of theprojected light spots in the first viewfinder image; and automaticallyfocusing a digital camera lens associated with the digital camera usinga closest one of the calculated triangulation measure of distances. 2.The method of claim 1, further comprising: capturing a second referenceimage of the target subject without projected light spots proximate intime to the capturing of the first viewfinder image; electronicallyregistering the first and second images; electronically generating adifference image of pixel values obtained by subtracting pixel valuesfrom the registered first and second images; and electronicallyidentifying light spot locations associated with the projected lightspots in the difference image, wherein the programmatically calculatingstep uses the identified spot locations in the difference image tocalculate the triangulation measures of distance.
 3. The method of claim2, wherein the electronically identifying step uses pixel color toidentify spot location.
 4. The method of claim 2, wherein the digitalcamera is held in a cellular telephone and comprises a circuitconfigured to capture the first and second images within less than about0.067 seconds of each other, and wherein the auto-focusing step isconfigured to move the lens to a focused position within about 0.033seconds after the first image is obtained to carry out the automaticallyfocusing step.
 5. The method of claim 1, wherein during the videorecording mode, the method further comprises electronically patchinglight spots in video recording images using pixel data fromcorresponding locations in at least one reference viewfinder image thatis devoid of light spots during video streaming to provide videorecording images without light spots.
 6. The method of claim 1, whereinthe auto focusing step uses a camera to subject distance “u” that iscalculated according to:u=fs/np where “n” represents a displacement measured in number of pixelsin position between a light spot in the image and a center of the image,“f” is a focal length of the camera lens, “s” is a distance between thecamera lens and an emitter axis for emitter(s) used to generate theprojected light spots, and “p” is pixel pitch.
 7. The method of claim 1,wherein the projecting step comprises emitting light from a light sourcein a visible part of a light spectrum.
 8. The method of claim 2, whereinthe capturing the first and second image steps are carried out using thesame exposure setting and the same white balance setting.
 9. The methodof claim 1, wherein the projecting the light spots step is carried outby substantially concurrently projecting a plurality of discretehorizontally spaced apart spots.
 10. The method of claim 2, furthercomprising automatically electronically correlating a region of interestin the first and second images to thereby provide image stabilization.11. A method of auto-focusing a digital camera in a cellular telephonein still image and/or video recording mode, comprising: projecting aplurality of spatially discrete light spots toward a target subjectusing the cellular telephone digital camera in a viewfinder mode;capturing a viewfinder image of the target subject having light spots inresponse to the projecting step; programmatically determining spotpositions in the image; calculating a triangulation measure of distancefor each of at least a plurality of the light spots in the image;selecting a focus distance for the camera based on a closest calculatedspot distance; and automatically focusing a digital camera lens in thecellular telephone digital camera using the selected focus distance toobtain a second image.
 12. The method of claim 11, wherein the selectingstep selects the focus distance from a correlation table definingdifferent focal lengths relative to pixel location of a light spotassociated with the closest spot distance in the spot image.
 13. Themethod of claim 11, further comprising: registering the viewfinder andsecond images; and electronically subtracting corresponding pixel valuesin the images to generate a difference image with the spots against adark background.
 14. The method of claim 13, wherein the electronicallysubtracting step is repeated for all pixels in a region of interest. 15.The method of claim 13, wherein the electronically subtracting step isconfigured to consider pixels of different colors.
 16. The method ofclaim 11, wherein, in the video recording mode, the method is carriedout by substantially continuously repeating the steps of projecting,capturing, determining and automatically focusing, and wherein themethod further comprises: electronically patching light spots in videorecording images with pixels from corresponding spot locations in atleast one reference viewfinder image, the at least one reference imagebeing devoid of light spots to thereby provide video recording imageswithout light spots.
 17. A method of auto-focusing a digital camerahaving a lens in a cellular telephone in still image and/or videorecording mode, comprising: (a) projecting a plurality of spatiallydiscrete light spots toward a target subject using the cellulartelephone digital camera; (b) capturing a first image of the targetsubject having the light spots in response to the projecting step; (c)electronically calculating a triangulation measure of distance for eachof a plurality of the light spots in the captured first image; and (d)programmatically focusing the camera using a focus distance of thetarget subject based on the triangulation measure of distance having alowest distance value.
 18. A method according to claim 17, wherein thecapturing step is carried out with QVGA resolution in viewfinder mode.19. A method according to claim 17, wherein the programmaticallyfocusing step is carried out using a camera to subject distance “u” thatis calculated according to:u=fs/np where “n” represents a displacement measured in number of pixelsin position between the spot in the image and a center of the image, “f”is a focal length of the camera lens, “s” is a distance between thecamera lens and an emitter axis for emitter(s) used to generate theprojected light spots, and “p” is pixel pitch.
 20. A method according toclaim 17, wherein the electronically calculating step comprises applyinga polynomial function to pixel values representing the spot image in adirection of interest.
 21. A method according to claim 17, furthercomprising video recording images using the cellular telephone camerawhile repeating steps (a)-(d), wherein the method further comprises:electronically patching light spots in video recording images usingpixel data from at least one reference viewfinder image that is devoidof light spots during video streaming to provide video recording imageswithout light spots.